Large payload externalization support #459

YunchuWang · 2025-08-13T15:41:22Z

This pull request introduces support for externalized payload storage using Azure Blob Storage, enabling large payloads to be stored out-of-band and referenced via tokens. The changes add a new abstraction for payload storage, implement an Azure Blob-based store, and provide integration points for both client and worker dependency injection to enable this feature. Additionally, a new data converter is introduced to automatically externalize large payloads based on configurable thresholds.

Externalized Payload Storage Infrastructure:

Added the IPayloadStore interface to abstract storing and retrieving large payloads out-of-band, with async upload and download methods.
Implemented BlobPayloadStore, an IPayloadStore that stores payloads as compressed blobs in Azure Blob Storage and returns opaque tokens for retrieval.
Introduced LargePayloadStorageOptions to configure externalized storage (enable/disable, threshold, connection string, container name).

Data Conversion Enhancements:

Added LargePayloadDataConverter, a DataConverter that wraps another converter and externalizes payloads exceeding a configurable size threshold, using the configured IPayloadStore.
Updated the base DataConverter class to support external storage by adding the UsesExternalStorage property and clarifying documentation.

Dependency Injection & Configuration:

Added UseExternalizedPayloads extension methods to both IDurableTaskClientBuilder and IDurableTaskWorkerBuilder to register the required services and wrap the configured data converter for externalized payload support. [1] [2]
Registered new dependencies and package references for Azure Blob Storage and required system annotations. [1] [2] [3]

These changes collectively enable seamless, configurable support for handling large payloads in Durable Task workflows by storing them externally and referencing them within orchestration messages.

YunchuWang · 2025-08-13T20:20:05Z

sample validated
info: Microsoft.DurableTask[1]
Durable Task gRPC worker starting and connecting to uksautop-etbbheebdcgm.uksouth.durabletask.io.
info: Microsoft.Hosting.Lifetime[0]
Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
Content root path: D:\projects\durabletask-dotnet\samples\LargePayloadConsoleApp
info: Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient[40]
Scheduling new LargeInputEcho orchestration with instance ID '3745aa067f8a4a3bb48c40b8b4e8093a' and 71 bytes of input data.
info: Microsoft.DurableTask[4]
Sidecar work-item streaming connection established.
Started orchestration with direct large input. Instance: 3745aa067f8a4a3bb48c40b8b4e8093a
info: Microsoft.DurableTask.Client.Grpc.GrpcDurableTaskClient[43]
Waiting for instance '3745aa067f8a4a3bb48c40b8b4e8093a' to complete, fail, or terminate.
info: Microsoft.DurableTask.Worker.Orchestrations[600]
'LargeInputEcho' orchestration with ID '3745aa067f8a4a3bb48c40b8b4e8093a' started.
info: Microsoft.DurableTask.Worker.Activities[603]
'Echo' activity of orchestration ID '3745aa067f8a4a3bb48c40b8b4e8093a' started.
info: Microsoft.DurableTask.Worker.Activities[604]
'Echo' activity of orchestration ID '3745aa067f8a4a3bb48c40b8b4e8093a' completed.
'Echo' activity of orchestration ID '3745aa067f8a4a3bb48c40b8b4e8093a' completed.
'Echo' activity of orchestration ID '3745aa067f8a4a3bb48c40b8b4e8093a' completed.
'Echo' activity of orchestration ID '3745aa067f8a4a3bb48c40b8b4e8093a' completed.
'Echo' activity of orchestration ID '3745aa067f8a4a3bb48c40b8b4e8093a' completed.
'Echo' activity of orchestration ID '3745aa067f8a4a3bb48c40b8b4e8093a' completed.
'Echo' activity of orchestration ID '3745aa067f8a4a3bb48c40b8b4e8093a' completed.
'Echo' activity of orchestration ID '3745aa067f8a4a3bb48c40b8b4e8093a' completed.
'Echo' activity of orchestration ID '3745aa067f8a4a3bb48c40b8b4e8093a' completed.
'Echo' activity of orchestration ID '3745aa067f8a4a3bb48c40b8b4e8093a' completed.
'Echo' activity of orchestration ID '3745aa067f8a4a3bb48c40b8b4e8093a' completed.
'Echo' activity of orchestration ID '3745aa067f8a4a3bb48c40b8b4e8093a' completed.
info: Microsoft.DurableTask.Worker.Orchestrations[601]
'LargeInputEcho' orchestration with ID '3745aa067f8a4a3bb48c40b8b4e8093a' completed.
RuntimeStatus: Completed
UsesExternalStorage (result converter): True
SerializedInput: dts:v1:payloadtest:2025/08/13/20/16/51/e3afe72b95ce49f493a34bd32e1b1b52
SerializedOutput: dts:v1:payloadtest:2025/08/13/20/17/02/16290bdc18f94a5caeb9e37ee5afc517
Deserialized input equals original: True
Deserialized output equals original: True
Deserialized input length: 1048576

…/durabletask-dotnet into wangbill/large-payload

src/Abstractions/Converters/BlobPayloadStore.cs

src/Abstractions/Converters/LargePayloadStorageOptions.cs

test/Grpc.IntegrationTests/LargePayloadTests.cs

YunchuWang · 2025-08-19T17:24:00Z

TODO: when load test, getting 503 busy periodically. should add retry when upload/download from blob
This orchestration failed with the following error:

Microsoft.DurableTask.TaskFailedException: Task 'SayHello' (#8) failed with an unhandled exception: Ingress is over the account limit. RequestId:7f562149-b01e-0044-3d2b-116c5b000000 Time:2025-08-19T17:03:34.3967120Z Status: 503 (Ingress is over the account limit.) ErrorCode: ServerBusy Content: ServerBusyIngress is over the account limit. RequestId:7f562149-b01e-0044-3d2b-116c5b000000 Time:2025-08-19T17:03:34.3967120Z Headers: Server: Microsoft-HTTPAPI/2.0 x-ms-request-id: 7f562149-b01e-0044-3d2b-116c5b000000 x-ms-client-request-id: 6583fc0c-16ac-4d93-baa7-f7fb9bc1f3cb x-ms-error-code: ServerBusy Date: Tue, 19 Aug 2025 17:03:34 GMT Content-Length: 213 Content-Type: application/xml

Stack trace:

at Microsoft.DurableTask.Worker.Shims.TaskOrchestrationContextWrapper.CallActivityAsync[T](TaskName name, Object input, TaskOptions options)
at AspNetWebApp.Scenarios.HelloActivities.RunAsync(TaskOrchestrationContext context, ActivityPayload payload) in /src/Orchestrations/HelloActivities.cs:line 18
at Microsoft.DurableTask.TaskOrchestrator`2.Microsoft.DurableTask.ITaskOrchestrator.RunAsync(TaskOrchestrationContext context, Object input)
at Microsoft.DurableTask.Worker.Shims.TaskOrchestrationShim.Execute(OrchestrationContext innerContext, String rawInput)

YunchuWang · 2025-08-20T03:20:04Z

perf test, 8mb, 100 rps, avg orch latency 30s

YunchuWang · 2025-08-20T14:54:38Z

YunchuWang · 2025-08-20T15:30:06Z

YunchuWang · 2025-08-20T15:39:21Z

tested scheduler backend

cgillum

This is a great feature, and a nice use of the DataConverter abstraction. Here's my first round of feedback.

src/Abstractions/DataConverter.cs

src/Abstractions/Abstractions.csproj

src/Abstractions/Converters/IPayloadStore.cs

src/Abstractions/Converters/BlobPayloadStore.cs

src/Abstractions/Converters/LargePayloadStorageOptions.cs

src/Abstractions/Converters/BlobPayloadStore.cs

src/Client/Core/DependencyInjection/DurableTaskClientBuilderExtensions.cs

test/Grpc.IntegrationTests/LargePayloadTests.cs

cgillum · 2025-08-22T22:39:33Z

test/Grpc.IntegrationTests/LargePayloadTests.cs

+
+namespace Microsoft.DurableTask.Grpc.Tests;
+
+public class LargePayloadTests(ITestOutputHelper output, GrpcSidecarFixture sidecarFixture) : IntegrationTestBase(output, sidecarFixture)


We need tests for entities as well. Specifically, entity operations and entity state needs to be considered.

I see samples, but I don't see automated tests for entities. Did I miss it?

sorry for confusion @cgillum i answered this in other comments. i found the current testing backend setup does not support entity, so i added it in the sample and validated it e2e. the entity sample e2e result looks correct. can we use sample for verification and later add this to dts e2e tests which more closely simulate the production experience?

src/Client/Core/DependencyInjection/DurableTaskClientBuilderExtensions.cs

src/Extensions/AzureBlobPayloads/Converters/BlobPayloadStore.cs

src/Extensions/AzureBlobPayloads/Converters/LargePayloadDataConverter.cs

cgillum · 2025-09-04T23:51:10Z

src/Extensions/AzureBlobPayloads/Converters/LargePayloadDataConverter.cs

+
+        // Upload synchronously in this context by blocking on async. SDK call sites already run on threadpool.
+        byte[] bytes = this.utf8.GetBytes(json);
+        string token = this.payLoadStore.UploadAsync(bytes, CancellationToken.None).GetAwaiter().GetResult();


It's very problematic for us to call async code from a non-async abstraction. It can create very bad performance and availability problems for the customer, like thread starvation. We should look into supporting async serialize and deserialize methods from the DataConverter abstraction.

src/Extensions/AzureBlobPayloads/Converters/LargePayloadDataConverter.cs

src/Extensions/AzureBlobPayloads/AzureBlobPayloads.csproj

src/Worker/Core/DependencyInjection/DurableTaskWorkerBuilderExtensions.cs

cgillum · 2025-09-04T23:57:49Z

test/Grpc.IntegrationTests/LargePayloadTests.cs

+
+namespace Microsoft.DurableTask.Grpc.Tests;
+
+public class LargePayloadTests(ITestOutputHelper output, GrpcSidecarFixture sidecarFixture) : IntegrationTestBase(output, sidecarFixture)


I see samples, but I don't see automated tests for entities. Did I miss it?

…/durabletask-dotnet into wangbill/large-payload

YunchuWang · 2025-09-17T01:06:58Z

abandoning this PR and move this feature change to another pr #468

YunchuWang added 4 commits August 12, 2025 14:24

save

5c80501

tests

560ecab

add sample

0f3e624

precise to second

d2b514f

YunchuWang requested review from halspang and torosent August 13, 2025 20:20

Merge branch 'main' into wangbill/large-payload

a565bb3

YunchuWang marked this pull request as ready for review August 13, 2025 20:20

Merge branch 'wangbill/large-payload' of https://github.com/microsoft…

4ec3f0c

…/durabletask-dotnet into wangbill/large-payload

YunchuWang requested a review from cgillum August 13, 2025 20:59

nytian reviewed Aug 18, 2025

View reviewed changes

src/Abstractions/Converters/BlobPayloadStore.cs Outdated Show resolved Hide resolved

src/Abstractions/Converters/LargePayloadStorageOptions.cs Outdated Show resolved Hide resolved

test/Grpc.IntegrationTests/LargePayloadTests.cs Show resolved Hide resolved

Merge branch 'main' into wangbill/large-payload

8575113

cgillum requested changes Aug 22, 2025

View reviewed changes

YunchuWang added 7 commits August 27, 2025 16:00

some fb

9d2bfa3

rename dts to blob

ef0961a

some fb

6387ad2

enabled always

b4527cc

split package

9168dc9

update sample

b5bedd0

remove enablestorage

6cebf0e

YunchuWang requested a review from nytian September 1, 2025 16:47

YunchuWang added 3 commits September 1, 2025 09:50

comment

ce51c0d

testname update

ecf89de

add entity sample for largepayload

3bd6c9a

cgillum reviewed Sep 4, 2025

View reviewed changes

YunchuWang and others added 15 commits September 5, 2025 17:12

some fb

20d3e8f

fix

e653680

fb

499c281

fb

e841e5e

fb

f182442

fb

e484f42

test

fb6d3fb

enable compression

f734b1b

update sample

50a210e

add gzip encoding header to detect for decompression

833406f

initial add async versions ser/deser

5ebe0b0

retry on blob upload/download

db91bf6

Merge branch 'main' into wangbill/large-payload

6a51f62

update calllers

c1a87b9

Merge branch 'wangbill/large-payload' of https://github.com/microsoft…

3244c10

…/durabletask-dotnet into wangbill/large-payload

YunchuWang mentioned this pull request Sep 12, 2025

support for async get custom status on orchestration Azure/durabletask#1238

Closed

YunchuWang added 11 commits September 13, 2025 09:38

continue updating async

4d4aeba

more update

62dda3e

update all async

4f0e5e3

enhance

af7a4c8

disallow sync calls when largepayload enabled

281fed2

more

52ec85c

more

a5549a3

refactor

738892b

more

1064329

more

cd05d6a

more

b3cbb43

YunchuWang closed this Sep 17, 2025


		namespace Microsoft.DurableTask.Grpc.Tests;

		public class LargePayloadTests(ITestOutputHelper output, GrpcSidecarFixture sidecarFixture) : IntegrationTestBase(output, sidecarFixture)

Large payload externalization support #459

Large payload externalization support #459

Uh oh!

Conversation

YunchuWang commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

YunchuWang commented Aug 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

YunchuWang commented Aug 19, 2025

Uh oh!

YunchuWang commented Aug 20, 2025

Uh oh!

YunchuWang commented Aug 20, 2025

Uh oh!

YunchuWang commented Aug 20, 2025

Uh oh!

YunchuWang commented Aug 20, 2025

Uh oh!

cgillum left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cgillum Aug 22, 2025

Choose a reason for hiding this comment

Uh oh!

YunchuWang Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

cgillum Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

YunchuWang Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cgillum Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cgillum Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

YunchuWang commented Sep 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

YunchuWang commented Aug 13, 2025 •

edited

Loading

YunchuWang Sep 5, 2025 •

edited

Loading